AppUcation No. 10/766,319 MS#306171.01/MFCP.153380 

Response Filed 09/03/2010 

Reply to Office Action of 06/24/2010 

AMENDMENTS TO THE CLAIMS 

Prior to the present communication, claims 10, 11, 15, 17, 18, 20-22, 24-28 and 
30-27 were pending in the subject application. Claims 10, 20, 22, and 27-28 have been amended 
and claim 21 has been canceled herein. Thus, claims 10, 11, 15, 17, 18, 20, 22, 24-28 and 30-27 
remain pending. This Listing of Claims will replace all prior versions, and listings, of claims in 
the application. 

Listing of Claims 

1-9. (Canceled) 

10. (Currently Amended) A computer-implemented method of capturing and 
treating content using a computer system having a processor, memory, and data storage 
subsystems, the computer-implemented method comprising: 

a) setting a mode of operation to a content capture mode for interpreting 
user input for the purpose of selecting an on-screen region of a display, and 
receiving a path drawn by a user, the path defining boundaries of the selected on- 
screen region of the display, wherein pixels comprising one or more graphical 
elements representing a first set of one or more textual characters are displayed in 
the selected on-screen region; 

b) capturing the pixels displayed within the selected on-screen region, and 
storing the captured pixels in an image file; 

c) receiving a user command to set the mode of operation to an annotation 
mode for interpreting the user input for the purpose of annotating the one or more 
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k switching the mode of 



operation to m [[the]] annotation mode in response to a user commzind; [[£ind]] 

d) receiving an annotation drawn by the user on the display, wherein the 
received annotation is implemented using a plurality of tools via a toolbar, the 
toolbar appearing after the selecting an on-screen region; [[and]] 

[[d)]] obtaining context information for the one or more graphical 
elements by automatically applying text recognition to the annotation, and storing 
the results of the text recognition as context information via the computer system, 
and 

f) obtaining additional context information by extracting the first set of 
one or more textual characters and extracting a second set of textuzil characters 
displayed in proximity with the first set, 

wherein the context information and the additional context information are 
[[is]] automatically stored in association with the image file. 

11. (Previously Presented) The computer-implemented method 



determining a window associated with the selected on-screen region; 

retrieving an application interface having a Uniform Resource Identifier 
(URI) property from the determined window or a parent window of the 
determined window; and 

obtaining the URI property as additional context information, the 
additional context information being automatically stored in association with the 
image file. 



according to claim 10, further comprising: 
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12-14. (Canceled) 

15. (Previously Presented) The computer-implemented method of claim 
10, further comprising: 

creating and storing a linking structure as the association between the 
image file and the context information. 

16. (Canceled) 

17. (Previously Presented) The computer-implemented method of claim 
15, wherein the linking structure is incorporated in a file separate from the stored image file and 
the stored context information. 

18. (Previously Presented) The computer- implemented method of claim 
15, wherein the linking structure includes at least one pointer pointing to the stored image file or 
the stored context information. 

19. (Canceled) 

20. (Currently Amended) The computer-implemented method of claim 10, 
wherein the context information is stored in such a manner as to be accessible to the user for 
performing at least one of the following: 

searching for said context information, 

displaying the context information simultaneously with the captured image 
pixels, and navigating a network to a source of the captured image pixels. 

21. (Canceled) 
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22. (Currently Amended) The computer-implemented method of claim 10, 
wherein the selected on-screen region is part of a displayed textual region, £ind the graphical 
elements comprise a first set of one or more textual characters displayed in the textual region, 
and the method further comprising: 

obtaining additional context information based on a second set of one or 
more textual characters displayed in the textual region, the additional context 
information being automatically stored in association with the image file ; and 

obtaining a first pointer between the context information and the textual 
region, and obtaining a second pointer between the additional context information 
and the textual region, wherein the first pointer differs from the second pointer . 

23. (Canceled) 

24. (Previously Presented) The computer-implemented method of claim 
27, further comprising: 

digitizing movements of a stylus across the display in order to receive an 
annotation; and 

obtaining additional context information based on the received annotation, 
the additional context information being automatically stored in association with 
the image file. 

25. (Previously Presented) The computer-implemented method of claim 
10, wherein the selected on-screen region includes at least a portion of a displayed web page or 
document, and the method further comprises: 
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using an application programming interface (API) to query an application 
for additional context information, the addition^ context information being 
automatically stored in association with the image file, the queried application 

causing the one or more graphical elements to be displayed. 

26. (Previously Presented) The computer-implemented method of claim 
25, further comprising: obtaining a uniform resource identifier (URI) of the web page or 
document as the context information, the URI being obtained as a result of the query using the 
API. 

27. (Currently Amended) A computer-implemented method of context 
harvesting from selected content using a computer system having a processor, memory, and data 
storage subsystems, the method: 

receiving a path drawn on a [[the]] display via by-a user via an input 
device , the drawn path defining [[the]] boundaries of a selected on-screen region 
of the display, the selected on-screen region comprising a plurality of pixels, 
wherein a displayed content of the selected on-screen region includes including 
befe textual data and underlying data comprising at least one of: an executable 
object, a file, or a link to remote content; 

capturing the plurality of pixels of the on-screen region; 

storing the captured pixels as an image file; 

in response to automatically determining that the displayed content of the 
on screen region includes the textual data, automatically extracting a character or 
word from the textual data and extracting complete sentences based upon 
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punctuation as context information in response to determining that the displayed 
content of the on-screen region includes the textual data via the computer system; 

pointing a first pointer from the context information to the displayed 
content; 

in response to determining that the displayed content of the on screen 
region includes the underljdng data, automatically extracting a property of the 
underlying data as additional context information in response to determining that 
the displayed content of the on-screen region includes the underlying data via the 
computer system, the property comprising at least one of: a file name, a file 
identifier, a uniform resource locator (URL), a uniform resource identifier (URI), 
a folder name, or meta-data; [[and]] 

pointing a second pointer from the additional context information to the 
displayed content; and 

storing the extracted context information and additional context 
information in association with the image file via the data storage subsystem, such 
that the context information is accessible when viewing the image file. 

28. (Currently Amended) A computer-implemented method of context 
harvesting from selected content using a computer system having a processor, memory, and data 
storage subsystems, the method comprising : 

receiving a display of content comprising at least one of textual data, an 
executable object, a file, or a link to remote content; 

setting a mode of operation to a content capture mode for interpreting 
input from an input device for the purpose of selecting an on screen region of a 
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display, and receiving a path drawn on the display by the input device, the drawn 
path defining the boundaries of a selected on screen region of the display, the 
selected on screen region comprising a plurality of pixels ; 

receiving a path drawn on the display which defines boundaries of a 
selected on-screen region of the content via user input; 

capturing the plurality of pixels of the on screen region ; 

capturing and storing the selected content captured pixels as an image file 
via the data storage subsystem, wherein the content displayed within the on 
screen region includes at least one of: textual data, an executable object, a file, or 
a linlc to remote content; 

switching the mode of operation to an annotation mode and receiving an 
annotation drawn on the display via the input device ; 

interpreting the input and annotating the content displayed within the on 
screen region; 

receiving an annotation on the display via the user input; 
performing text recognition on the received annotation as context 
information; 

pointing a first pointer from the selected content to the context 
information; 

automatically deteiTnining via the computer system that the content 
displayed within the on-screen region includes at least one of textual data and 
underlying data comprising at least one of an executable object, a file, and a linlc 
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automatically extracting as additional context information: 

a bounded region comprising a character or word from tiie textual 

data determined to be included in the on-screen region, and determining if the 
bounded character or word lies completely within the bounded selected on screen 
region, and 

a property of tiie underlying data determined to be included in the 
on-screen region, the property comprising at least one of: a file name, a file 
identifier, a uniform resource locator (URL), a uniform resource identifier (URI), 
a folder name, or meta-data; [[and]] 

pointing a second pointer from the selected content to the additional 
context information; and 

storing the context information and the additional context information in 
association with the image file via the data storage subsystem, such that the 
context information is accessible when viewing the image file. 

29. (Canceled) 

30. (Previously Presented) The computer-implemented method of claim 
10, wherein the annotation is stored as originally drawn as additional context information in 
association with the image file. 

31. (Previously Presented) The computer-implemented method of claim 
28, wherein the annotation is stored as originally drawn as additional context information in 
association with the image file. 
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32. (Previously Presented) The computer-implemented method of claim 
10, further comprising a link to the one or more graphical elements of the selected on-screen 
region, the link comprising one of an implicit link hidden behind the one or more graphical 
elements and an explicit link shown in conjunction with the one or more graphical elements. 

33. (Previously Presented) The computer-implemented method of claim 
10, further comprising a separate linking table or database entry utilizing one or more pointers to 
the one or more graphical elements. 

34. (Previously Presented) The computer-implemented method of claim 
10, further comprising displaying the obtained context information to the user as one of 
displaying a property in a window and hovering over the one or more graphical elements. 

35. (Previously Presented) The computer-implemented method of claim 
10, wherein the one or more graphical elements comprise underlying data, underlying objects, 
and underlying associations with remote content. 

36. (Previously Presented) The computer-implemented method of claim 
21, further comprising: determining that the second sets of textual characters reside within the 
path drawn by the user and any subsequent sets of textual characters reside outside the path 
drawn by the user. 

37. (Previously Presented) The computer- implemented method of claim 
10, wherein the received annotation is implemented using a plurality of tools via a toolbar, the 
toolbar appearing after a stylus has been lifted off the display. 
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